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Cu For a Markov chain both the detailed balance condition and the cycle 

^^ Kolmogorov condition are algebraic binomials. This remark suggests to study 

^^ reversible Markov chains with the tool of Algebraic Statistics, such as toric 

^^ statistical models. One of the results of this study in an algebraic parameteri- 

C^ zation of reversible Markov transitions and their invariant probability. 

H 

^ 1. Introduction. On a finite state space V we consider q-reversible (quasi- 

reversible) Markov matrices, i.e. Markov matrices P with elements denoted Pv^w, 
v,w eV, such that Pv^w = if, and only if, Pw^v = 0, v / w. 

The support of P is the graph ^ = {V, S'), where vw, v / w, is an edge if, and 

i only if, Pv^w and P^^v are both positive. We associate to each edge vw the two 

directed arcs v — )• w and w — )• v to get a directed graph without loops (i.e. arcs from 

h>. V to v) that we denote by & = {V,£/), see Figure 1. Theneighborhoodof v isN(v), 

CN the degree of v is d{v), the set of arcs leaving v is out(v), the set of arcs entering v 

00 isin(v). 

tH- Viceversa, given a connected graph '^ = {V,£'), a. q-reversible Markov matrix 

[^ P with structure ^ is such that Py^w = iivw ^ £' and Pv^w = if, and only if, 

O Pw^v = 0, vw G (f . Such a Markov matrix is characterized as a mapping P: iz/ — )• 

O M^ such that L„gN(v) Pv^w < 1, v G V. As we assume Py^w = 0, vvv ^ (f , we have 

Pv^v = 1 — LweN(v) Pv^w The support of P will be a sub-graph of ^. We are going 
. ;^ to need this detailed classification in the following. For the time being, we note 

^ that the setoff 'sis parameterized by a product of symplexes Xvev'5'(N(v)), where 

S{I) = [x gW : Y^ieiXi < 1 } denotes the (solid) simplex over the index set /. 

A Markov matrix P onV satisfies the detailed balance conditions if there exists 
K-(v) > 0, V G V, such that 

k:(v)Pv^w = K-(w)P„,^v, v,wGV. 
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Fig L Running example: undirected graph f# and directed graph & 

It follows that P is q-reversible and that 7i{v) = K:{v)/Y,vev '^(v) i^ ^^i invariant 
probability with full support. Equivalently, the Markov chain (X„), « = 0, 1,..., 
with invariant probability % and transition matrix [Py^w], v,w G V, has reversible 
bivariate joint distribution 



(1) P(X„=v,X„ 



+1 



w 



P(X„ = w,X„ 



+1 



VjWGV, n>Q. 



Such a Markov chain, and its transition matrix are called reversible. Reversible 
Markov Chains (MCs) are relevant in Statistical Physics, e.g. in the theory of en- 
tropy production, and in Applied Probability, e.g. the simulation method Monte 
Carlo Markov Chain (MCMC). The main aim of this paper is to find useful param- 
eterizations of the reversible Markov matrices of a given structural graph. 

In Section 2 we review some basics from Dobrushin, Sukhov and Fritts (1988), 
Kelly (1979), (Strook, 2005, Ch 5), Diaconis and Rolles (2006), Hastings (1970), 
Peskun (1973), Liu (2008). 

In Section 3 we discuss the algebraic theory prompted by the detailed bal- 
ance condition. The results pertain to the area of Algebraic Statistics, see e.g. 
Pistone, Riccomagno and Wynn (2001), Drton, Sturmfels and Sullivant (2009), 
Gibilisco et al. (2010). Previous results in the same algebraic spirit were presented 
in Suomela (1979) and Mitrophanov (2004). We believe the results here are new; 
some proofs depend on classical notions of graph theory that are reviwed in some 
detail because of our particular context. 

The discussion and the conclusions are briefly presented in Section 4. 

2. Background. 

2.1. Reversible Markov process. The reversibility of the bivariate joint dis- 
tribution in Equation (1) gives a second parameterization of reversible Markov 
chains. In fact, the stochastic process (X„)^>q with state space V is 2-reversible 
and 2-stationary if, and only if. Equation (1) holds. When the process is a Markov 
chain, the distribution depends on the bivariate distributions only. In particular, the 
process is 1-stationary: by summing over w E V, we have 



P(X„=v) = P(X„+i 



7r(v), veV,n>0. 
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Let V2 be the set of all subset of V of cardinality 2. The elements of V2 are the 
edges of the full graph on V. The following parameterization of the 2-dimensional 
distributions has been used in Diaconis and Rolles (2006): 

0., = p(x„ = v,x„+i=v), vev, 

0W = P {Xn = V,X„+i = w) + P (X„ = W,Xn+l = V) 
= 2P{Xn=V,Xn+l=w), VWGV2. 

The number of parameters isA'^+(2) = ( J); moreover it holds 

1 = £ P(X„ = v,X„+i = w) = £ 0.,+ £ d^, 



v.wEV 



vev 



VWEV2 



hence = (0v,, 0yj belongs to A{V U V2), where A(/) = {x£R' : Y^iei^i = 1 } de- 
notes the (flat) simplex on the index set /. 

Given an undirected graph ^ = {V, <§) such that P (X„ = v,X„+i = w) = if vvv ^ 
S, then the vector of parameters 6 = {dy, Qg) belongs to the convex set A(V U S). 
We note that the vertices V are identified with loops of the transitions because 
0v = P(X„ = v,X„+i=v). 

The marginal probability 7i can be written using the Q parameters: 



or, in matrix form, 



w] 



wev 



ev + ^ X ^^ 



' M'eN(i') 



where T is the incidence matrix of the graph ^. 

Example 1 (Running example). Consider the graph ^ = (V, (f) with V 
{1,2,3,4} and (o = {12,23,34, 14,24 1, see left side of Figure 1. Here 



1 

2 
3 
4 



12 23 34 14 24 
1 1 0" 
110 1 
110 
111 



All 71 's are obtained this way if we admit positive probability on loops: 



G. PISTONE AND M.P. ROGANTIN/ ALGEBRA OF REVERSIBLE MC 

Proposition!. 1. The map 



y. A(yu^)9 e 



0^ 



71= [iv ^r] 



dv 



GA(y) 



is a surjective Markov map. 
2. The image of (0, Bg), Bg G A((f ), is the convex hull of the half points of each 
edge of the simplex A(y) whose vertices are connected in ^ . 

Proof. 1. Each probability % is the image of = (71,0^). 

2. If By = 0, then % = Lwgn(v) \^vwBvw, where Lwgn(v) fe = 1 and T^v is the 

vw-column of T. Hence, ^T^w is the the middle point of the v- and the w- 

vertex of the simplex A(y ). 

D 

Item 1 of the proposition leaves open the question of the existence of an element 
B such that Bg > for each tt > 0. This is discussed in the next subsection. Item 
2 shows that, while all tt's can be obtained if loops are allowed {Bv > 0), only a 
convex subset of A(y) is obtained if we do not allow for loops (By = 0). 

2.2. From a positive 71 to positive transitions. Given 71, the fiber Y^^(7i) is 
contained in an affine space parallel to the subspace 

ker(/y + Ir) = 1 G M^^'^ : By + ^-TBg = 

Each fiber 7^^(71), tt > 0, contains special solutions. The solution {71, Qg) is not of 
interest because we want Bg > 0. If the graph has full connections, ^ = (V, V2), 
there is the independence solution Bv = 7r(v)^, Bm = 27l{v)7l{w). e If 7r(v) > 0, 
V G y, a strictly positive solution is obtained as follows. Let d(v) be the degree of 
the vertex v and define a transition probability by A(v,w) = l/{2d{v)) if vvv G (f, 
A(v,v) = 1/2, and A{v,w) = otherwise. A is the transition matrix of a random 
walk on the graph ^, stopped with probability 1/2. Define a probability on V x V 
with Q{v,w) = 7r(v)A(v,H'). If Q{v,w) = Q{w,v), v,w G V, we are done: we have 
found a 2-reversible probability with marginal 71 and such that Q{v,w) > if, and 
only if vw G (f . Otherwise, if Q{v,w) / Q{w,v) for some v, w G V, we turn to the 
following Hastings-Metropolis construction. 

Proposition 2. Let Qbea probability onV xV such that Q{v, w) >Oif and 
only if vw e S or V = w. Write 7l{v) = Y.„ 2(v, w). Given f :]0, 1 [x]0, 1 [->]0, 1 [ a 
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symmetric function such that f{x,y) < xAy, then 



P{v,w) 



if{Qiv,w),Q{w,v)) ifv^w, 

\n{v)-Zw:w^vP{v,w) ifv = w. 



is a 2 -reversible probability onV xV such that 7l{v) = Y,wP{v,w) and P{v,w) > 
if, and only if vw G S". 

Proof. For vvv G (f we have P{v, w) = P{w, v) > 0, otherwise zero. As P{v, w) < 
Q{v, w), V / w, it follows 

P{v,v) = n{v)- £ P{v,w) 

>£e(v,w)- £ Q{v,w) 

= e(v,v)>o. 

We have L,v/'(v, w) = n(y) by construction and, in particular, P{v, w) is a probabil- 
ity on y X y. n 

Remark 1. 1. The proposition applies to 

(a) f{x,y) =xAy. This is the standard Hastings choice. 

(b) f{x,y) = xy/{x+y). This was suggested by Barker 

(c) f{x,y) = xy. In fact, as y < 1, we have xy < x. 

2. Given a joint probability P, the corresponding parameters 

dvw = '2P{v, w) and dv = P{v, v) 

are strictly positive for vw £ (^ and v £V, otherwise zero. We have shown 
the existence of a mapping from n in the interior o/A(y) to a vector of 
parameters 6 in the interior ofA{V U S'). 

2.3. Parameterization of reversible Markov matrices. An q-reversible Markov 
matrix P supported on a graph ^ is parameterised by its non-zero extradiagonal 
values Pv^w, i-e. by the elements of Xvgv/'S'°(N(v)), where S° denotes the open 
solid simplex. As W is connected, the invariant probability of the Markov matrix 
P is unique, therefore the joint 2-distribution is uniquely defined. If moreover the 
Markov matrix is reversible, the joint 2-distribution is symmetric and the 6 param- 
eters are computed. Viceversa, given the 0's, the transition matrix is given by 

,^. p _P{Xn=V,X„+i=w) _ Oyw 



P {Xn — v) 2dv + LzgZ)(v) % 
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The mapping 6 i— ;• (Pv^w '■ (v — )• w) G £/) is a rational mapping and the number of 
degrees of freedom is #V + #<#' — 1. 

Denoting Idy + Lz6D(v) ^ by k{v), from (2), the detailed balance conditions 
follow 

andl^,,K:(v) = 1. 

2.4. A reversible Markov matrix is an auto-adjoint operator The detailed bal- 
ance condition 7i{v)Py^w = 7i{w)Pw^v is equivalent to P being adjoint as a linear 
operator onL^(7r) 

{Pf,g)^ = {f,Pg)^, f,geL^{n), 

where {Pf,g)„ = Lv{L.Pv^.J{w))g{v)K{v). 

As L^{'Ji) is isomorphic to the canonical Euclidian space '^ via the linear map- 
ping 

/: L^{k) 9 / ^ diag(7r)i/V e M^, 

the Markov matrix P is mapped to the symmetric matrix 

S = IoPor^ =diag(7r)'/2/'diag(7r)"^/2. 

This implies that a reversible Markov matrix is diagonalizable; in particular the 
left eigenvector n and the right eigenvector I oi P are both mapped to the eigen- 
vector (7r(v)i/^ : V G y) of 5. 

Each element of the symmetric matrix S is positive if, and only if, the corre- 
sponding element of P is positive. Each reversible Markov matrix P with invariant 
probability 7i is parameterized by a unique symmetric matrix S. 

Viceversa, let ^(v, w) = s(w, v) > be defined for vw G (f . Extend s to all couples 
V / w, vw ^ (f by s(y,w) = 0. If 



(3) ^ s{v,w)^/^<^/^, veV, 

we can define 

5(v,v) = ^l=(y^-I^(v,w)y^^) >0 
V7r(v) V v^v J 

to get a symmetric non-negative matrix S = [s{v,w)], v,w G V. The matrix P 
diag 7r^^/^5diag tt^/^ has non-negative entries; is a Markov matrix because 

Y^Pv^u, = Y^n{v)-'/'s{v,w)n{wy/^ = 1, veV; 




G. PISTONE AND M.P. ROGANTIN/ ALGEBRA OF REVERSIBLE MC 7 

satisfies tlie detailed balance equations 

7r(v)Pv^„,, = K{vyl'^K{wYl'^s{v,w) = n{w)P„^y. 

We can rephrase the computations above as follows. 

Proposition 3. The set of Markov matrices which have structure ^ and are 
reversible with invariant probability 71 is parameterized by 

--n{vY'^l'^n{wyl'^s{v,w), {v^w)^£/, 

1 - LwGN(v)P„^„, 

the poly tope of all positive weight functions s: S ^ M>o and (unnormalized) pos- 
itive probability n satisfying the inequalities (3). 

2.5. Kolmogorov's theorem. Let ^ = (V, <f) be a connected graph. For each 
closed path co, that is a path on the graph such that the last vertex coincides 
with the first one, CO = vqVi . . . v„vo, we denote by r{o)) the reversed path r{(o) = 
voVn ■ ■ .viVq. The Kolmogorov's characterization of reversibility based on closed 
paths is well known. However, we give here a variation of the proof by Suomela 
(1979) as it is an introduction to the algebraic arguments in the next Section. The 
proof has been modified to allow null transitions. If 7 = vqVi • • • v„_iv is any path 
connecting vq to v we write P^ to denote the product of transitions along 7, i.e. 

Theorem 1 (Kolmogorov's theorem). The Markov irreducible matrix P is 
reversible if and only if for all closed path ft) 



(4) 



3r(o) 



Proof. Assume that the process is reversible. By multiplying together all de- 
tailed balance equations 

k:(v;)Pv,^v,+i = K-(v,-+i)Pv,^i^v,, / = 0, 1, . . .n,v„+i = Vo, 

and clearing the K"'s we obtain (4). 

Viceversa, assume that all closed path have property (4). Fix once for all a ver- 
tex Vo and consider a generic path 7 from vq to v. First we prove that there exists a 
positive constant k{v), depending only on v, such that P^ = k{v)P''^^\ In fact, for 
any other path / = vqVj . . . v'j,v with the same endpoints vq and v, 7r(7') is a closed 
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path. Denoting by k'{v) the corresponding constant, Kolmogorov's condition im- 
pUes k{v) = k'{v). Moreover, for any vertex w connected with v, consider the path 
yw and the corresponding constant k{w). We have: 

r r\,^n.r\v^v ^ k(y)r ry^jf„r„^y 

i.e. the detailed balance condition on w and v. D 

In the next Section we will discuss the algebraic interpretation of Kolmogorov 
condition. 

3. Algebraic theory. The present section is devoted to the algebraic struc- 
ture implied by the Kolmogorov's theorem for reversible Markov chains. We refer 
mainly to the textbooks Berge (1985) and Bollobas (1998) for graph theory, and 
to the textbooks Cox, Little and O'Shea (1997) and Kreuzer and Robbiano (2000) 
for computational commutative algebra. The theory of toric ideals is treated in de- 
tail in Sturmfels (1996) and Bigatti and Robbiano (2001). General references for 
algebraic methods in Stochastics are e.g. Drton, Sturmfels and Sullivant (2009), 
Gibilisco et al. (2010). The relevance of Graver bases, see Sturmfels (1996), has 
been pointed out to us by Shmuel Onn has in view of the applications discussed in 
De Loera et al. (2008) and Onn (to appear). 

3.1. Kolmogorov's ideal. We denote by ^ = (V, (^) an undirected graph. We 
split each edge into two opposite arcs to get a connected directed graph (without 
loops) denoted by ^ = (V, .s/). The arc going from vertex v to vertex w is denoted 
by V — >• w or (v — )■ w). The graph & is such that (v — )■ v) ^ ^2/ and (v — )• w) G =2/ if, 
and only if, (w — >• v) G s^ . Because of our application to Markov chains, we want 
two arcs on each edge, as it was explained in Figure 1 . 

The reversed arc is the image of the 1 -to- 1 function r: .s^ ^ s^ defined by r (v — )• 
w) = (w — ;■ v). Kpath is a sequence of vertices (O = vqVi • • • v„ such that (v^_i — )• 
Vk) € £^ , k = l,...,n. The reversed path is denoted by r{o)) = v„v„_i • • • vq. Equiv- 
alently, a path is a sequence of inter-connected arcs (0 = ai.. Mn, at = (v^-i — )• v;t). 
and r{(o) = r(a„) . . .r(ai). 

A closed path CO = vqv\ ■ ■ ■ v„-i vo is any path going from a vertex vo to itself; 
r{(o) = vov„_i • • -viVo is the reversed closed path. In a closed path any vertex can 
be the initial and final vertex. If we do not distinguish any initial vertex, the equiv- 
alence class of paths is called a circuit. A closed path is elementary if it has no 
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2 
-►f 




i i 

II (I k^i 



Fig 2. r/ie 6 c>'c/ei- o/a graph: ©^ = (1 ^ 2)(2 ^ 4)(4 -^ 1), (Ob = (2 ^ 3)(3 ^ 4)(4 ^ 2) 
COc = (1 ^ 2)(2 ^ 3)(3 ^ 2)(4 ^ 1), oJd = r(a)A), (H£ = r{aB),(OF = r{(Oc). 



proper sub-closed-path, i.e. if does not meet twice the same vertex except the ini- 
tial one vq. The circuit of an elementary closed path is a cycle. We denote by ^ the 
set of cycles of &. 

Consider the commutative indeterminates P = [Pv^w]. (v — )■ w) G =e/, and the 
polynomial ring k[Pv^„ : (v — ;■ w) G -s^], i.e. the set of all polynomials in the inde- 
terminates P and coefficients in the number field k. 

For each path (O = a\---an,ak^ -^Z, ^ = 1 ,...,«, we define the monomial term 



co = ar--an^P'^ = Y\Pa^. 

k=i 

For each a G £/, let Na{o)) be the number of traversals of the arc a by the path co. 
Hence, 

p(0 _ T-f pN„{co) 

Note that ft) i— ;■ /"" is a representation of the non-commutative concatenation of 
arcs on the commutative product of indeterminates. Two closed paths associated 
to the same circuit are mapped to the same monomial term because they have the 
same traversal counts. The monomial term of a cycle is square-free because no arc 
is traversed twice. 

Figure 2 presents the 6 cycles in the running example of Figure 1. This list 
of cycles is larger than a basis of cycles in the undirected graph ^, for instance 
{(Oa,(Ob}. We will see below that all directed cycles are needed for the algebraic 
argument. 

Definition 1 (K-ideal). The Kolmogorov's ideal or K-ideal of the graph ^ is 
the ideal of the ring k[Py^^. : (v — t- w) G .s^] generated by the binomials P^ — P''^'^', 
where (0 is any circuit. The K-variety is the k-affine variety of the K-ideal. 

Our main application concerns the real case fc = M, but the combinatorial struc- 
ture of the K-ideal does not depend on the choice of a specific field. A interesting 
choice for computations could be the Galois field fc = Z2. 
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Proposition 4 (Examples of K-ideals). Let the Markov matrix P with struc- 
ture 'S he reversible. 

1. The real vector Pv^w, (v — t- w) G £^, is a point of the intersection of the 
variety of the K-ideal with S{2) = Xvgv5'(v), where 

5(v) = U G Mf (^'^ : £ P.(w)<l|. 

I. aGout(v) J 

2. Let (Xn)n>Q be the stationary Markov chain with transition P. Then the real 
vector of joint probabilities p{v,w) = P(X„ = m,X„+i = v), (v — )■ w) G s^, is 
a point in the intersection of the K-variety and the simplex 

S{£^) = J p G M^ : £ P{a) <\\. 
y aes^ J 

Proof. 1. It is the first part of the Kolmogorov's theorem. 

2. Let (0 = vq... v„vq be a closed path. If 7i is the stationary probability, by 
multiplying the Kolmogorov's equations by the product of the initial proba- 
bilities at each transition, we obtain 

;r(vo)7r(vi) • • • 7l{v„)P„g^v^ ■ ■■Pv„^vo = 71{vq)k{v„) ■ ■ ■ 7r(v„)Pvo^v„ • • -Pvi^vo, 

hence 

p(vo,Vi)p{vi,V2) ■ ■ ■ piVn,VQ) = /^(vo, V„)p(v„, V„_i) • • •/^(vi , Vq). 

However, in this case the Kolmogorov's equations are trivially satisfied as p{v, w) 

p{w,v). D 

The K-ideal has a finite basis because of the Hilbert's basis theorem. Precisely, 
a finite basis is obtained by restricting to cycles, which are finite in number. We un- 
derline that here we consider all the cycles, not just a generating set of cycles. The 
related result by (Mitrophanov, 2004, Th. 1) is discussed in the next subsection. 

Proposition 5 (Cycle basis of the K-ideal). The K-ideal is generated by the 
set of binomials P® — P'"(®), where (0 is cycle. 

Proof. Let CO = vqvi • • • vq be a closed path which is not elementary and con- 
sider the least k > I such that Vk = v^i for some k' < k. Then the sub-path coi 
between the ^'-th vertex and the ^-th vertex is an elementary closed path and the 
residual path (O2 = vo- --VkiVk+i ■■■vq is closed and shorter than the original one. 
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The arcs of co are in 1-to-l correspondence with the arcs of coi and (1)2, hence 
Na{(o) = Na{(0\) +Na{(ih), ^ € -Sif . The procedure can be iterated and stops in a fi- 
nite number of steps. Hence, given any closed path (O, there exists a finite sequence 
of cycles (0\,. . .,(0u such that the list of arcs in (O is partitioned into the lists of arcs 
of the «,'s. From P®' - /""('*'') = 0, / = 1 , . . . , Z, it follows 



i=\ (=1 



pw ^ TTpffl' = TTpK®/) = pKffl) 



D 



The K-ideal is generated by a finite set of binomials and this set has the same 
number of elements as the set of undirected cycles of ^. If the graph is a tree, the 
cycles reduce to those of the type vi — )• V2 — )• vi and the binomials are identities, 
hence the K-ideal is equal the the full ring. This is an involved way to prove all 
stationary Markov process on a tree is reversible (Kelly, 1979, Lemma 1.5). 

The cycle basis of Proposition 5 belongs to the special class of bases, namely 
Grobner bases. We refer to the textbooks Cox, Little and O'Shea (1997) and Kreuzer 
and Robbiano (2000) for a detailed discussion. We review the basic definitions of 
this theory, which is based on the existence of a monomial order )^, i.e. a total order 
on monomial terms which is compatible with the product. Given such an order, the 
leading term LT(/) of the polynomial / is defined. A generating set is a Grobner 
basis if the set of leading terms of the ideal is generated by the leading terms of 
monomials of the generating set. A Grobner basis is reduced if the coefficient of 
the leading term of each element of the basis is 1 and no monomial in any element 
of the basis is in the ideal generated by the leading terms of the other element of 
the basis. The Grobner basis property depends on the monomial order. However, a 
generating set is said to be a universal Grobner basis if it is a Grobner basis for all 
monomial orders. 

The finite algorithm for testing the Grobner basis property depends on the defi- 
nition of syzygy- The syzygy of two polynomial / and g is the polynomial 



gcd(LT(/),LTte))^ gcd(LT(/),LTU))"- 

A generating set of an ideal is a Grobner basis if, and only if, it contains the 
syzygy Syz(/,g) whenever it contains the polynomials / and g, see (Cox, Little 
and O'Shea, 1997, Ch 6) or (Kreuzer and Robbiano, 2000, Th. 2.4. 1 p. 1 1 1). 

Proposition 6 (Universal G-basis). The cycle basis of the K-ideal is a re- the? 
duced universal Grobner basis. 
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Proof. Choose any monomial order >~ and let coi and o>2 be two cycles with 
COi >- r{(Oi), i = 1,2. Assume first they do not have any arc in common. In such a 
case gcd(P'"i ,P'^) = 1 and the syzygy is 

SyzfP"'' — /'''(^i) P'^ _pr(ah)\ — 

pah/pWi _pr{mi)\ _ pcoupmi _pr(a>2)\ _ p(0ipr{<O2) _ pr(mi) pCOi 

which belongs to the K-ideal. 

Let now a be the common part, that is gcd(f®',/"^) = P". The syzygy of 

p(Oi _ pr{(Oi) ^jj^ poh _ piioh) js 

pmi-apr((i>2) _ p(02- a pr((Oi) _ pra /pcai-apr{a>2)-ra _ pO)2-apr{mi)-ra\ 

which again belongs to the K-ideal because coi — a + r{co2 — a) is a cycle. In fact 
COi — a and (Oz — cc have in common the extreme vertices, corresponding to the 
extreme vertices of a. Notice that a is the common part of (Oi and o>2 only if it 
is traversed in the same direction by the both cycle. The previous proof does not 
depend on the choice of the leading term of the binomials, therefore the Grobner 
basis is universal. The Grobner basis is reduced because no monomial of a cycle 
can divide a monomial of a different cycle. D 

Example 2 (Running example continue). Figure 3 is an illustration of the 
proof. 

On this example we can see why the two (Oa and (Oc do not generate the K-ideal. 
In fact, from 

P\^2P2^aPa^i=Pi^aPa^2P2^i 

Pl^2P2^3P3^4P4^l = Pl^4P4^3P3^2P2^1 



it follows 



P4^lA^2^2^lA^4 {P2^3P3^4P4^2 — P2^4P4^3P3^2) — 



which, in turn, gives the binomial of (Ob if P^^\P\_^2P2^\P\^4 7^ and, therefore, 
the factor p"p''W can be cleared. This is confirmed by the use of a symbolic alge- 
braic software such as GoCcA, see CoCoATeam (online). This computation shows 
that the (Og equation does not belong to the ideal generated by the (Oa , (Oc equa- 
tions unless we add the condition P4^iPi^2^2^i A^4 7^ 0. Notice that COa and (Oc 
are the cycles obtained from the spanning tree 3 — ?• 4 — t- 1 — t- 2. 
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rfcoj 



r.(A-'A)-' 



r(a> 



naj 



Fig 3. Running example: illustration of the proof of Proposition 6 



Example 3 (Running example: Monomial basis of the quotient ring). Take 
any order on vertexes, e.g. 1 -< 2 -< 3 -< 4, and derive a lexicographic order on 
arcs: 1^2-<l->4^2-^l-<2->3-<2->4^3^2^3^4^4-^l-<4^ 
2 ^ 4 — )■ 3. Take the same order on indeterminate s Pa, a G si , and the lexicographic 
order on monomials. We check that the leading terms of the there binomials in the 
G-basis are p''y'"A) ^fias) ^pr(coc) ^ ^^^ Figure 2. The exponents of the leading terms 
of the G-basis are 

si 1^2 1^4 2^1 2^3 2^4 3^2 3^4 4^1 4^2 4^3 



A^W^a)) 





1 


1 

















1 





N{r{(aB)) 














1 


1 











1 


N{r{(iic)) 





1 


1 








1 











1 



Each monomial P^ = Ylaes^/ Pa"' ^ ^ '^> > '■* reduced by the K-ideal to a monomial 
whose exponent does not contain any of the counts in the table. E.g. P\^i\P2^\P3^2 
is an element of the monomial basis of the polynomial ring mod the K-ideal. 

3.2. Cycle and cocycle spaces. We adapt to our context some standard tools of 
algebraic graph theory, namely the cycle an cocycle spaces, see e.g. (Berge, 1985, 
Ch 2) and (Bollobas, 1998, II.3). 

Let 'lo be the set of cycles. For each cycle ft) € '^ we define the cycle vector of 
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Fig 4. Example of cocycle vector. All arcs not shown take value 0. 



(0 to be z{(o) = {Za{(o) '■ a G £^), where 

Za{(0) -- 



+1 if a is an arc of «, 
— 1 if r(fl) is an arc of ft), 
otherwise. 



Note that ZrUi) (^) = —Za{(0). If z+ and z are the positive and the negative part of 
z, respectively, then z^{(0) = Na{(o) and z^{(o) = Na{r{(o)). It follows that P™ = 

pN{co) ^ pzHco) ^ Yi^^^ii^'^^) and 

(5) p(o _pr[(o) ^pz+{(o) _pz-{m)^ 

More generally, the definition can be is extended to any circuit ft) by defining 

Za{(0)=Na{(0)-Nr[a){(0). 

The equality z+(ftj) = N{(o) holds if, and only if, a G ft) implies r{a) ^ (0,a ^ s^ . 

Let Z(i^) be the cycle space, i.e. the vector space generated in W^ by the cycle 
vectors. 

For each proper subset B of the set of vertices, 7^ B C V we define the cocycle 
vector of B to be u{B) = {ua{B) : a G s^), with 



Ua{B) 



+ 1 if a exits from S, 
— 1 if a enters into B, 
otherwise. 



a G 



See an example in Figure 4. Note that Mr(Q)(5) = —Ua{B). 

Let L'^(^) be the cocycle space, i.e. the vector space generated in R*^ by the 
cocycle vectors. Let U be the matrix whose rows are the cocycle vectors u{B), 
(b^B 'ZV. The matrix U = [ua{B)]ih^BcyMesi' is the cocycle matrix. 

The cycle space and the cocycle space are orthogonal in W^ . In fact, for each 
cycle vector z{(o) and cocycle vector u{B), we have 

Zr[a){(0)Ur(a){B) = {-Za{(0)){-Ua{B)) = Za{CO)Ua{B), fl G ^, 
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SO that 



Zico) ■ u{B) = Y, Zai(0)UaiB) = Y, Za{(0)Ua{B) + £ Zo(co)m«(5) 
aGi/ a&co r(a)ea) 



2Yza{(0)Ua{B)=2 



a€W 



I 1- I 1 

aem,u„{B)=+l aG(a,«fl(B)=-l 



It is shown e.g. in the previous references that the cycle space is the orthogonal 
complement of the cocycle space for undirected graphs. In our setting it is the 
orthogonal complement relative to the subspace of vectors x such that a,.(„) = — x^. 
As we are interested in elements of the cycle space with integer entries, i.e. those 
elements z = (z^ : a S -2^) of the cycle space that can be exponents of a monomial 
P~ = Ylaes/ Pq" ' we are going to use the following matrix encoding of our problem. 

Definition 2 (Model matrix). Consider the matrix E = [Ee^a]ee<^,aes^ whose 
element Ee^a in position {e,a) is 1 if the arc a is une of the directions of the edge e, 
zero otherwise. Let U be the cocycle matrix. The model martix is the block matrix 

E 

U 

It follows Z{!3) = kerA. A lattice basis of the lattice Z(^) = kerA n Z-^ is a linear 
basis o/kerA with integer entries. 

The matrix A has dimension ^S + W — \. In fact E can be re-arranged as 
[l#s\l#s\, with l#g the identity matrix, and U has #V — 1 hnearly independent rows, 
the dimension of the cocycle space. 

Example 4 (Running example continue). Table 1 shows the matrix A and a 
lattice basis of'L{&) computed with GoGcA. The top matrix is the E matrix; the bot- 
tom matrix is the U matrix, where three linearly independent rows are highlighted. 
The two bottom row vectors are the lattice basis. 

3.3. Toric ideal. We want to show that the K-ideal is the toric ideal of the 
model matrix A, see Definition 2. Basic definitions and theory are in (Sturmfels, 
1996, Ch 4), see also Bigatti and Robbiano (2001). 

Consider the polynomial ring Q[Pa: a £ £/] and the Laurent polynomial ring 
Q[f^VB ^ : e e ^,0 ^B CV], together with their homomorphism h defined by 

(6) h:Pa^ll'e"Wf'^='^''''^- 
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Table 1 
Running example. Model matrix A and a basis ofZ{&). 





1^2 


1^4 


2^3 


2^4 


3^4 


2^ 1 


4^ 1 


3^2 


4^2 


4^3 


u 


1 














1 














14 





1 














1 











23 








1 














1 








24 











1 














1 





34 














1 














1 


{1} 


1 


1 











1 


1 











{2} 


-1 





1 


1 





1 





-1 


-1 





{3} 








1 





1 








1 





1 


{4} 





-1 





-1 


-1 





1 





1 


1 


{12} 





1 


1 


1 








1 


1 


1 





{13} 


1 


1 


-1 





1 


-1 


-1 


1 





-1 


{14} 


1 








-1 


-1 


-1 








1 


1 


{23} 


-1 








1 


1 


1 








-1 


-1 


{24} 


-1 


-1 


1 





-1 


1 


1 


-1 





1 


{34} 





-1 


-1 


-1 








1 


1 


1 





[123} 





1 





1 


1 





-1 





-1 


-1 


[124} 








1 





-1 








-1 





1 


[134} 


1 





-1 


-1 





-1 





1 


1 





[234} 


-1 


-1 











1 


1 












z{03a) 

z{o}b) 



1 


-1 





1 





-1 


1 





-1 











1 


-1 


1 








-1 


1 


-1 
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As Ee^a = ^e,r(a) fof ^H edge € and arc a, the first factor in (6) is a symmetric 
function s{v,w) = He^e'^' " = ■5(>v>v). We could write 

(7) /i:n^,^^(v,w)nC""''''- 

B 

The kernel /(A) of h is called the tone ideal of A, 

/(A) = {/ e Qi^a -.aes^]: /{t^^"^ : a G i/) = o} . 
The toric ideal /(A) is a prime ideal and the binomials 

P'^-P' , zeZ-"^, Az = 0, 

are a generating set of /(A) as a Q- vector space. A finite generating set of the ideal 
is formed by selecting a finite subset of such binomials. The basis we find is a 
Graver basis. 

We recall the definition of Graver basis as it is given in De Loera et al. (2008) 
and we apply it to the cycle lattice Z(^). We introduce a partial order and its set 
of minimal elements as follows. 

Definition 3 (Graver basis). Let zi and zi he two element of the cycle lattice 

Z(^). 

1. z\ is conformal to Z2, Zi E Zi, if the component-wise product is non-negative 
(i.e. zi and Z2 cifs ifi the same quadrant) and \z\\ < |Z2| component-wise, i.e. 
Zi,aZ2,a >Oandz\,a <Z2,aforalla ££/. 

2. A Graver basis of7L{Q^ is the set of the minimal elements with respect to the 
conformity partial order C. 

Proposition 7. 

1. For each cycle vector z G Z(^), z = Y.ae'g ^i.^)z{.(>>)y A (ft)) G Q, there ex- 
ist cycles ft)i,. . . ,ft)„ G "^ and positive integers a(ft)i), . . . ,a(w„), such that 
Z'^ > Z^ (ft),-), z" > zr {(Oi), i= \,...,n, and 

n 

z = J^a{(0i)z{(0i). 
(=1 

2. The set {z(a)) : coe'rf} isa Graver basis ofZ{^). 
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Proof. 1. For all G) G "^ we have —z,{(o) = z{r{(o)), so that we can assume 
all the A(g))'s to be non-negative. Notice also that we can arrange things in 
such a way that at most one of the two direction of each cycle has a non-zero 
coefficient. We define 

£/+iz) = {ae£/:za>0}, £/-{z) = {ae£/:Za<0}, 

and consider two subgraph of ^ with a the set of arcs restricted to J2/+, £/^, 
respectively. We note that r^/^ (z) = ^_ (z) and r=g/_ (z) = ^+ (z) ; in particu- 
lar, both ^+(z), i/- (z) are not empty. We drop from now on the dependence 
on z for ease of notation. 

We show first there is a cycle whose arcs are in £/+. If not, if a cycle of full 
graph & has one arc in =2/+, it would exists vertex v such that out(v) n =2/+ = 
while in(v) (!£/+ / 0. Let u{v) be the cocycle vector of {v}; we derive a 
contradiction to the assumption z • m(v) =0. In fact, 

Z-u(y)= ^ ZaUaiv)+ ^ ZaUa{v) 

because each of the terms ZaUa{v), a G £/+, is either or equal to —Za < if 
a G in(v). 

Let (Oi be a cycle in £/+ and define an integer a(ft)i) > 1 such that z^ — 
a(cOi)z^(ft)i) > and it is zero for at least one a. The vector z.^ = z — 
a(wi)z(G)i)belongs to the cycle space Z(i^), and moreover =g/+(z^) C ^+(z). 
By repeating the same step a finite number of times we obtain a new rep- 
resentation of z in the form z = T!i=i Oi{c^i)z{(Oi) where the support of each 
a{(Oi)z'^ {(Oi) is contained in =2/+. It follows 

n n 

(8) z+ = 52a(a);)z+(w,-) and z" = 52a(wOz"(ft>;) 

i=l i=\ 

2. In the previous decomposition each z(g);), / = 1, ...,«, is conformal to z- In 
fact, from z+ > z+(ft>0 and z" > z"(ft>,), it follows ZaZa{(Oi) = z+z+(ft>,-) - 
ZaZai^i) > and |za(w;)| = zi {(Oi) - Za {(Oi) < z^ + z^ = \za\- Therefore 
z{(Oi) Q z. 

D 

Example 5 (Running example continue). We give an illustration of the pre- 
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Fig 6. Running example. Computation of the conformal representation of the z of Figure 5 



vious proof. Consider the cycle vectors 

1^2 2^1 2^3 3^2 3 
z{(Oa) = 



^44^3 4^1 1^4 2^44^2 



1 



1 



z{(Ob) = ( 
z(«c) = ( 1 




-1 















1 


1 


-1 


1 


-1 





1 


-1 


1 


-1 


1 



-1 







1 





and the element of the cycle space z. = z(g)a) + 2z{(0b) + 2z{(0c), see Figure 5. We 
have 

z=z(«a) + 2z(cob) + 2z(«c) = (3,-3,4,-4,4,-4,0,0,-1,1) 
z+= z+(«b)+3z+(«c) =(3,0,4,0,4,0,0,0,0,1) 

as it is illustrated in Figure 6. 

Theorem 2 (The K-ideal is toric). 

1. The K-ideal is the toric ideal of the matrix A. 

2. The binomials of the cycles form a Graver basis of the K-ideal. 

Proof. 1. For each cycle (O the cycle vector z{(o) belongs to Z(^). From 
Equation (5), P^^t'") -pz («) =p® -P'-(«), therefore the K-ideal is contained 
in the toric ideal 1{A) because of Proposition 5. 

To prove the equality we must show that each binomial in /(A) belongs to 
the K-ideal. From Proposition 7.1, it follows that 



pz+ _ pz = fT(pz^(ffl'))«(«') _ Y\{P^' ('*''))" 



[Oh) 



(=1 



1=1 
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belongs to the K-ideal. 
2. The Graver basis of a toric ideal is the set of binomials whose exponents are 
the positive and negative parts of a Graver basis of Z(^). From Propositions 
7 and the previous item the proof follows. 

D 

3.4. Non-zero K-ideal. The knowledge that the K-ideal is toric is relevant, be- 
cause the homomorphism definition in Equation (7) provides a parametric repre- 
sentation of the the variety. In particular, the strictly positive Pa, a ^ s^ , we given 
by: 

B 

(9) =s{v,w) W tB Y\ q\ 5(v,w)>0, tB>0. 

B: veB.w0 B: weB,v(^B 

We observe that the first set of parameters, s{v,w), is a function of the edge, 
while the second set of parameters, tB, represents the deviation from symmetry. In 
fact, as 



<w,v)nc-"^"^=^(v,w) n^. 



B 

2 




we have Pv^w = [IIb^b ) ^w^v, so that Pv^w = Pw^v if. and only if, 

(ntr'^^A =1, vev. 

As the rows of E are linearly independent, the s{v,wys parameters carry #S' 
degrees of freedom to represent a generic symmetric matrix. The second set of 
parameters is not identifiable because the rows of the U matrix are not linearly 
independent. The parameterization (9) can be used to derive an explicit form of 
the invariant probability. All properties of the parameterization are collected in the 
following Proposition. 

Theorem 3. Consider the strictly non-zero points on the K-variety. 

1. The symmetric parameters s[e), e £ (o, are uniquely determined in Equation 
(9). The parameters fg, 7^ S C V are confounded by kerU = {U't = 0} 

2. An identifiable parameterization is obtained by taking a subset of parameters 
corresponding to linearly independent rows, denoted by tB, B C o5^.' 

(10) p,^vv=^(v,w) n ^fi U ^b' 

Bey : veB.wiB B<zy' : weB,\4B 
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3. The detailed balance equations, K'(v)Pv^vv = ^{w)Pw^v, cif^ verified by 

(11) ^(V)oc Yl t^^ 

B: veB 

Proof. 1. We have logP = E^s + U't for P = (P„^h, : (v -^ w) G ^), s = 
{s{e) : eeS'),t = {tB:Q^S<ZV). liE'si + U'ti =E's2 + U't2, then£'^(5i - 
S2) = because the rows oiE are orthogonal to the rows of U. Hence, si = S2 
because E has full rank. Finally, U'ti = U't2. 

2. The sub-matrix of A formed by E and by the rows oiU in y has full rank. 

3. Using Equations (9), we have: 



k{v)s{v,w) Y\ tB Y\ tj^^ = K{w)s{v,w) Yl tB Y{ tj, 



S : veSM^S S: weS,\4S S : wGS,v^5 S: veS-W^S 

which is equivalent to 



'^(v) n ^i=<^) n 'I 

S : veS.wiS S : weS^x^S 



By multiplying both terms in the equality by Os : ves wes^B' ^^ obtain 

'^(v) n ^i = <'') n ^B, 

S : veS S : weS 

so that k{v) = Yls. vesh^ depends only on v and satisfy the detailed balance 
condition. 

D 

We are now in the position of stating an algebraic version of Kolmogorov's 
theorem. 

Definition 4. The detailed balance ideal is the ideal of the ring 

Q[fc(v) : V G V,Pv^vv-, (v ^ w) G £/] 
generated by the polynomials 

vev 

K{v)Py^„ - k{w)P,^,„ (v ^ w) G £/. 

The first polynomial in the list of generators, IIvgv ^i^) ~ 1' assures that the k"'s 
are not zero. 
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Theorem 4. 

1. A point P = [Pv^w\y^„Q^ with non-zero components belongs to the variety 
of the K-ideal if, and only if there exists K = (k'(v) : v G V) such that {k,P) 
belongs to the variety of the detailed balance ideal. 

2. The detailed balance ideal is a toric ideal. 

3. The K-ideal is the K-elimination ideal of the detailed balance ideal. 

Proof. 1. One direction is the Kolmogorov's theorem. The other direction 
is a rephrasing of Item 3 of Theorem 3. 

2. This ideal is the kernel of the homomorphism defined by (7), i.e. Pv^w ' — > 

^(v,w)nBC'^' together with k{v) ^ Ub. veBte^- 

3. The elimination ideal is generated by dropping the parametric equations of 
the indeterminates to be eliminated. 

D 

3.5. Parameterization of reversible transitions. The parameterization in The- 
orem 3.2 is to be compared to that in Proposition 3. Both split the parameter space 
into a representation of the invariant probability and a generic symmetric function. 
The latter is a special case of the former. In fact, it is obtained by the use of the 
basis of the cocycle space where each B is the set containing one vertex. 

The parameterization of reversible Markov matrices is obtained by adding to 
the representation in Equation (10) the relevant inequalities. Let us check first the 
degrees of freedom. As a reversible Markov matrix supported by a connected graph 
W is parameterized by the joint 2-distributions of the stationary Markov chain, the 
number of degrees of freedom is #V + #(f — 1, i.e. the cyclotomic number of the 
graph ^. In the parameterization of Proposition 3 the probability n carries #V — I 
degrees of freedom, therefore s must carry #S' degrees of freedom. 

Theorem 5. Let P be a matrix with supporting graph ^ = (V, S"). Let S^ be 
a family of subsets ofV such that the cocycle vectors ub, B G S^, span the coclycle 
space. 

L P is a reversible Markov matrix if and only if there exists a non-negative 
symmetric function s\ V x V — )• R>o which is supported on the edges S and 
there exist positive parameters ts > 0, B & 5^ , such that 

(12) Py^„ = s{v,w) n ^5 n ^b\ (v^w)g=2/. 

Bey : veB.wiB Bey : weB.y^B 

T 

and the invariant probability is proportional to K = YIbbv^b ■ 
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Bbv weN{v) Bbw 

3. The parameters s{e), e ^ S and k{v), v G V, are identifiable, while the pa- 
rameters tg, B € o5^, are identifiable if ug, B G ^, is a basis of the cocycle 
space. 



Proof. 1 . Follows from (10) and (11) and 

Y\s. veS^B 



k(wV/^k(v)-^/^ 



lis. weS^B 



n 



n 



Scy:veS,wi^S scy-.wes.v^s 

2. It follows from Equation (12) as P is a transition probability. 

3. Assume there exists two set of parameters t^'\ s^'\ i = 1,2 giving the same 
P, and define t = f f^^/fP), s = s^^^/s^^\ k{v) = K^^\v) / K^^\v) . It follows 



,-1 



(v — > w) G £/, 



i=s{v,w) n ^5 n 'B ' 

Be.y : veBMfB Bey : weB,Y0 

\=s{w,v) n ^5 n ^B^ (w^v)Gi/, 

Bey-. weBa^B Bey-. veB,wiB 



hence s{v,w)s{w,v) = s{v,wY = 1- In turn we get 



1= n ^^ n 'B 

\Bey:veB.w0 Bey-.weBA^B 
The identifiability of the fg's follows from 



-1 



k{w) 



logPv^v, =l0g5(v,w)+ ^ {\ogtB)UB 

Bey 



D 



Example 6 (Running example continue). An over-parameterization of two 
transition probabilities of the K -variety is: 

P3^4 = s{3,4) f{3} ?{i3} t{2,3} ^{1,23} ^{4} ^{1^4} ^{2,4} ^{1,2,4} 
Pa^S = ■5'(3,4) f{4| ?{i 4| t[2A} ^{1,2,4} ^{3} ^{1^3} ^{2.3} ^{L2,3} 

By choosing the cocycle basis oS^ = {{l},{3},{l,2}}, we have: 



(k(1] 



—2 —7 



K{3)=t^,] 

[ Jf (4) = 1 



?|3} = 'c(3)-^/2 
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The transition matrix parameterized by s{e), e G (o and t^, S £ J^ is 





1 




2 


3 


4 


1 

2 
3 
4 


5(1,2) f{i| 



_5(l,4)f{i}?{i 


2} 


5(1,2) ?^Y} 
5(2,3) f^-ljf{l,2} 

^(2,4) f{i,2} 




^(2'3)f{i|2}^{3} 
5(3,4) f{3} 


5(1,4) f^-if^-i^} 

5(2,4) f^-|2} 

5(3,4) ^-3^ 

•k 



where the diagonal terms are uniquely defined if the following set of inequalities is 
true 

(l>s{l,2)t{i,2}+s{lA) 

1 > 5(l,2)f{i}?{i,2} +^(2,3)f{3} +^(2,4) 
l>5(2,3)?{i,2} +5(3,4) 

^ 1 >5(l,4)f{i}?{i^2}+^(2,4)f{i,2}+5(3,4)?{3} 

4. Discussion. The algebraic analysis of statistical models of the type p oc 
t^ where A is an integer matrix has been first introduced in Geiger, Meek and 
Sturmfels (2006), where an implicit binomial form of the model is derived from 
the monomial form. 

Our analysis differs from that in two respect. First, we move backwards from the 
binomial form represented by the Kolmogorov's conditions to the monomial form. 
Second, we parameterize the transition probabilities, so that the normalization re- 
quires more than one constant. 

Our parameterization of a reversible Markov matrix is based on a generic weight 
function s{e) on the edges g G <# of the structure graph ^ and a monomial form 
of the invariant probability. When the basis of the cocycle space is given by the 
vertexes of the graph, the parameterization is identical to that the classical form 
Pv^w = 7i{v)^^^^7i{wy^^s{v,w). The monomial form of the unnormalized invari- 
ant probability K"(v) = YIbeSiS: vesh^ suggests the use of a family of sets ^ smaller 
than a cocycle basis y in order to get a parsimonious statistical model. For exam- 
ple, if the graph 1# is a square N x N grid, a coarse-grained model could use n xn 
sub-grids, I <n <N. 

One distinct advantage of the implicit binomial form is its ability to fully de- 
scribe the closure of its strictly positive part, i.e. the extended exponential family. 
The computation of an Hilbert basis of the non-negative integer kernel of the cocy- 
cle matrix U leads to a parameterization of the extended exponential family as in 
Malago and Pistone (2010), see also Rauh, Kahle and Ay (2009). However, in this 
case the border of the model appears to consist simply on the deletion of edges in 
the support graph. 
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It follows from general properties of toric ideals that a Graver basis is a univer- 
sal Grobner basis and that a universal Grobner basis is a Markov basis, Sturmfels 
(1996). The Markov basis property is related with the connectedness of random 
walks on the fibers of A, see Diaconis and Sturmfels (1998) and subsequent lit- 
erature on MCMC simulation. In this case it would be a simulation of a random 
reversible Markov matrix. 

Finally, the knowledge of a Graver basis for the K-ideal provides efficient algo- 
rithms for discrete optimization, see De Loera et al. (2008) and Onn (to appear). 
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